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i— < (54) Title: DEVICE CONTROL VIA IMAGE-BASED RECOGNITION 

(57) Abstract: A video device is coupled to an appliance controller, via an image-based user recognition application. When a 
particular user's face is recognized in the provided image, or sequence of images, a controller accesses a database containing a user 
profile corresponding to the recognized user. This user profile contains the user's preferences for presentation material, such as 
Q preferences for particular types of entertainment at particular limes of the day. Based on the information in the user profile, the 
controller activates one or more devices, such as a television set or stereo set, and selects material for presentation to the user that is 
likely to coincide with the users desires at that time. 
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order to effect this user-specific organization and presentation, the user typically identifies 
himself or herself to the control system, by providing a user name, or by presenting an 
identification device. Although user-identification is commonplace while using computer 
devices, or in particular environments, such as an office environment, user identification 
5 within a less structured environment, such as a home, may present problems. In an office 
environment, for example, employees can be expected to wear identification badges, or to 
purposefully engage biometric scanners, such as a facial recognition device, a retinal scanner, 
a fingerprint scanner, and so on. In a home environment, on the other hand, having to type in 
an identification sequence, or to carry an identification token, or to purposefully engage a 
10 recognition device, in order to activate or otherwise control an entertainment device would 
likely be viewed as an annoyance, or at least an inconvenience. 

BRIEF SUMMARY OF THE INVENTION 

It is an object of this invention to provide an unobtrusive means of identifying 

15 a user to facilitate the control in a home-automation system. It is a further object of this 
invention to provide an automated user-specific presentations of material based on a 
recognition of the user. It is a further object of this invention to provide automated television 
activation and channel selection, based on an automated recognition of a user. It is a further 
object of this invention to provide automated control of appliances, based on an automated 

20 recognition of a user. 

These objects and others are achieved by providing a video device that is 
coupled to an appliance controller, via an image-processing application. When a particular 
user is recognized in an image, a controller accesses a database containing a user profile 
corresponding to the recognized user. This user profile contains the user's preferences for 

25 presentation material, such as preferences for particular types of entertainment at particular 
times of the day. Based on the information in the user profile, the controller activates one or 
more devices, such as a television set or stereo set, and selects or recommends material for 
presentation to the user that is likely to coincide with the user's desires at that time. 

30 BRIEF DESCRIPTION OF THE DRAWINGS 

The invention is explained in further detail, and by way of example, with 
reference to the accompanying drawings wherein: 

FIG. 1 illustrates an example television controller with face recognition capabilities in 
accordance with this invention. 
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Copending U.S. patent application "REMOTE CONTROL PROGRAM 
SELECTION BY GENRE", U.S. serial number 09/282,319, filed 03/31/99 for Karen 
Travato, Dan Pelletier, Paul Rankin, and Jacquelyn Martino, Attorney Docket PHA 23,644, 
incorporated by reference herein, presents techniques for programming the channel-up and 
5 channel-down keys of a remote control based on genre, and also based on a user profile for 
particular preferred channels. 

In each of these aforementioned inventions, the system provides control, or 
recommendations, based on an identification of the current user or users of the device being 
controlled. As noted above, an 'active 1 method of user identification, such as the entry of a 

10 user name or code, or carrying an identification token, may not be consistent with a 

consumers view of "convenience". In accordance with this invention, this user identification 
is provided in a 'passive 1 manner, based on a recognition of the presence of the user in the 
vicinity of the device being controlled. An image capture device, such as a video camera, 
captures an image of a scene in the vicinity of the device being controlled, and a user- 

1 5 recognition application, such as a face-recognition application, identifies the user from the 
image. 

FIG. 1 illustrates an example device control system 100 with user-recognition 
capabilities in accordance with this invention. For ease of reference, a television system 130 
is presented as the device being controlled, although any other user-controllable device, such 

20 as a stereo system, an Internet access device, and so on may be the device being controlled. 
An imaging device 110, such as a video camera, captures an image of objects within its field 
of view 115, and provides this image to a user-recognizer 120. 

In a preferred embodiment, the user-recognizer 120 is a multi-stage face 
recognition device that searches the image for an area of the image having the general 

25 characteristics of a face, based on the location of flesh tones, the location of non-flesh tones 
corresponding to eye brows, demarcation lines corresponding to chins, nose, and so on (see 
the referenced US patent 5,835,616). If a face is detected, it is characterized for comparison 
with reference faces that are stored in a database associated with the face recognizer 120. 
This characterization of the face in the video image is preferably the same characterization 

30 process that is used to characterize the reference faces, and facilitates a comparison of faces 
based on characteristics, rather than an 'optical 1 match, thereby obviating the need to have 
two identical images (current face and reference face) in order to locate a match. In a 
preferred embodiment, the number of reference faces is relatively small, typically limited to 
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above. For example, a gait recognition device can be used to identify a person based on the 
motion of a person in the image. A less complex recognition device may be based on the 
user's size. For example, the user profiles may be segregated into 'adult 1 and 'child 1 profiles, 
wherein some or all controls are based on whether the user is an adult or child based on the 
5 user's size. In a less sophisticated embodiment, the recognition may merely be the presence of 
a person in a particular location, such as a user's favorite chair, using for example, detection 
via an infrared image collector. Other image-based techniques, such as color distribution, 
edge distribution, and the like, may also be used to facilitate the identification of users. These 
and other embodiments may also be used to 'pre-process' the image information to facilitate a 

10 rapid identification from the video image. Copending U.S. patent application, "USING A 

SECONDARY SENSOR FOR OPTIMIZED VIDEO COMMUNICATIONS", serial number 
09/634,682, filed 8/8/00 for Michael Bakhmutsky, Attorney Docket US000158, incorporated 
by reference herein, presents techniques for finding warm items, such as humans, in a video 
image, to optimize the encoding of specific portions of an image, based on a lower-cost, 

15 lower-resolution infrared image that corresponds to the video image. This same principle can 
be used to provide a preliminary user identification, which is further modified or confirmed 
based on a further processing of a video image. 

FIG. 2 illustrates an example flow diagram of a device control system with 
user recognition in accordance with this invention. Face recognition is used in FIG. 2 as the 

20 paradigm of image-based user recognition, for ease of understanding. At 210, the video 

image (from the imaging device 110 of FIG. 1) is scanned to determine the presence of a face 
in the image. This scanning may be continuous, periodic, event-driven, or otherwise initiated. 
For example, the scanning may be continuous until a recognized face is detected, then 
periodically scanned, for example, at intervals corresponding to the traditional 'program 

25 change' periods (in the U.S., programs generally change on half-hour or hour boundaries). 
Alternatively, the scanning may be coupled to a television, and activated at the beginning of 
each commercial break. 

If, at 215, a face is detected in the image, the characteristics of the detected 
face are determined, at 220. If a user has been identified, via a prior image scan, the 

30 characteristics of the detected face are compared to the characteristics of the identified user, 
at 225. If the detected face does not correspond to the identified user, or if there is no 
currently identified user, a database of user faces 201 is searched, at 230, for a user face 
having the same characteristics of the detected face. If, at 235, a user is identified that 
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202 includes multi-user profiles, such as the channels that are normally viewed when users A 
and B, but not C, are present, and so on. 

As would be evident to one of ordinary skill in the art, the control block 260 
and the user profile 202 may also operate using state-dependent rules. For example, if a 
5 particular user generally leaves and enters a room repeatedly, the rules may be structured to 
invoke a 'waiting period 1 before clearing the user's identification from the current set, at 250. 

The foregoing merely illustrates the principles of the invention. It will thus be 
appreciated that those skilled in the art will be able to devise various arrangements which, 

10 although not explicitly described or shown herein, embody the principles of the invention and 
are thus within its spirit and scope. For example, the control system 100 may be configured to 
operate with other control systems, or within other control system architectures, as well. For 
example, the devices 1 10-140 may be interconnected via a home automation network and 
control protocol. Home automation standards include IEEE 1394, X-10, HAVi, HomeAPI, 

1 5 Jini, and the like. IEEE 1 394 and X-l 0 are communication protocols; HAVi is a software 
architecture using IEEE 1394; Home API is an open industry specification that defines a 
standard set of software services and application programming interfaces which enable 
software applications to monitor and control home device; and Jini is a distributed software 
architecture (network) wherein clients see devices and services as objects. The television 

20 controller 140 may include, for example, a personal computer with an IEEE 1 394 interface to 
a HAVi-compliant satellite television receiver, and the face recognizer 120 is a software 
application that is also located within the personal computer. As would also be evident to one 
of ordinary skill in the art, the control of the device may be via the control an ancillary 
device, such as controlling the channel being viewed on a television by controlling a cable or 

25 satellite receiver system, a VCR tuner, and the like. These and other system configuration 
and optimization features will be evident to one of ordinary skill in the art in view of this 
disclosure, and are included within the scope of the following claims. 
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A device control system (100) comprising; image within a field of 

an imaging device (1 10) that is configured to collect an un g 

view (115) of the imaging device (110), on & 

a user recognizer (120) that is configured to identity 

recognitionofch— 

a device controller (140) that is configured to facilitate 

n nm nf Maim 1 , further including: 



2. 
and 



chmc «e riS «cs *e ^ «o «he « of— oh—., 
5 o The system (100) of claim 2, wherein 

includes at least one of : 

aplurality of face characteristics, 
a plurality of gait characteristics, 
a plurality of infrared characteristics, 
25 a plurality of color characteristics, and 

a plurality of size characteristics. 

The system (100) of claim 1, wherein 
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6. The system (100) of claim 1, wherein 

the device controller (140) effects the control of the device (130) via a home- 
automation network. 

5 

7. The system (100) of claim 1, wherein 

the device (130) is a television receiver, and 

the profile of preferences (202) includes preferred channels at specified times. 



10 8. The system (100) of claim 1, wherein 

the control of the device (130) includes at least one of: 
an activation of the device (130), and 
a tuning of the device (130). 

15 9. The system (100) of claim 1, wherein 

the user recognizer (120) is further configured to identify at least one other 
user, based on a recognition of characteristics of the at least one user within the image, and 

the device controller (140) is further configured to effect the control of the 
device (130) based on preferences associated with the at least one other user. 

20 

10. A method for controlling a device (130), comprising: 

recognizing (220) characteristics within an image, 

identifying a user based on a comparison (235) of the characteristics with a 
plurality of user characteristics (201), and 
25 facilitating control (260) of the device (130) based on one or more preferences 

associated with the user. 



1 1 . The method of claim 10, further including 

selecting the one or more preferences associated with the user from a plurality 
30 of preferences (202) that correspond to the plurality of user characteristics. 

12. The method of claim 1 1, wherein 

the device (130) is a television receiver, and 

the one or more preferences includes preferred channels at specified times. 
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1 3 The method of claim 10, wherein 

facilitating control (260) of the device (130) includes at least one of: 
activating the device (130), 

recommending one or more control settings of the device (130), 
presetting a control setting in a control device for the device (130), 



and 



tuning the device (130). 



10 14 The method of claim 10, further including: 

recognizing other characteristics within the image, 

identifying an other user based on a comparison of the other characteristics 
with the plurality of user characteristics (201), and 

further facilitating control of the device (130) based on one or more 
1 5 preferences associated with both the user and the other user. 

15. A television control system (1 00) comprising: 

a video camera (110) that provides an image, 

a face recognition device (120), operably coupled to the video camera (110), 

20 that identifies a user based on the image, 

a television controller (140), operably coupled to the face recognition device 
(120), that facilitates a control of the television (130) based on preferences associated with 
the user. 



25 16. 



The television control system (100) of claim 15, wherein 
the control of the television (130) includes at least one of: 
activating the television (130), 

recommending one or more settings of the television (130), 
presetting options of the television (130), and 
30 selecting a source of material for presentation on the television (130). 

17 The television control system (100) of claim 15, wherein 

the user recognition device is further configured to recognize multiple users 

based on the image, and 



WO 02/32136 PCT/EP01/11141 

12 

the television controller (140) is further configured to effect the control of the 
television (130) based on preferences associated with the multiple users. 
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(57) Abstract: A video device is coupled to an appliance controller, via an image-based user recognition application. When a 
particular user's face is recognized in the provided image, or sequence of images, a controller accesses a database containing a user 
profile corresponding to the" recognized user. This user profile contains the user's preferences lor presentation material, such as 
preferences for particular types of entertainment at particular limes of the day. Based on the information in the user profile, the 
controller activates one or more devices, such as a television set or stereo set, and selects material for presentation to the user that is 
likely to coincide with the user's desires at that lime. 
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